Mood and Text Analyses

Study description

The dataset we’re looking at today is a relatively unique one. So far, we have collected data from about 65 adolescent participants (with more coming in), who have either recently attempted suicide (Suicide Attempt), reported having suicidal ideation (Suicidal Ideation), or are part of a psychiatric control group (Psychiatric Control). We then put our in-house Effortless Assesment of Risk States app (EARS) onto their phones (both Android and iPhone), which collects a wide range of data from the sensors in the phone. One fairly unique feature of our app is that we have a keyboard logger that logs everything typed by the user. In addition to the passive mobile sensing, individuals will also, daily, get an Ecological Momentary Assessment (EMA) that asks them to rate their mood on a scale of 1-100.

In this document, we’re going to first explore the daily mood score, and then we’re going to see how daily mood is related to various features we can extract from the text data we’ve gathered from the adolescent’s phone.

Daily mood score distribution

Let’s go ahead and take a look at the distribution of what we’re trying to predict: daily reported mood.

With this graph, each individual has their own daily mood score distribution (as indicated by color). This is a density plot, so wherever the density goes up means that there’s more data points within that range. For instance, the participant with the distribution in red that’s closest seems to have some major spikes at 60, 80, 90, and 100. That means there’s more mood ratings there than at other points, such as 75. Lookng at all the participants, it looks like there’s quite a bit of variability in reported mood from person to person. However, it might be useful to break it down even further and look at whether there are any differencs in distributions based on psychiatric group.

From what I can tell, it looks like there’s quite a bit of difference when looking at the psychiatric conrol vs the suicide attempt and suicide ideation group, but probobaly not much of a difference between suicide attempt and suicidal ideation. Let’s run a quick analysis to see if that’s the case. We’ll use the dabestr package, which is helpful for visualizing effects between different groups.

Alright, it looks as though my hunch was somewhat correct. This graph depicts in the top half all of the data points within the different groups. On the bottom half, the center line displays the mean daily mood of the psychiatric control as 0 (think of it as a baseline). The two sideways distributions are simply a measure of whether or not the mean is statistically significantly different from the psychiatric control. If the vertical black line for one of the groups crosses over the horizontal zero line (like the suicide ideation group), the difference in means is not statistically different. Daily reported mood in the suicide attempt group is significantly different from the psychiatric control group, but the suicidal ideation group is not siginficantly different from the psychiatric control group. Additionally, as anticipated, there is no significant difference between the suicidal ideation and suicide attempt group.

Now, these differences are rather interesting, but I’m curious in something a little bit different. For that, we’ll have to get everyone on the same playing field. There still seems to be a wide range of how people rate their mood on any given day–in essence, it’s possible that one person’s 75 could be another person’s 50. It might be more interesting to look, instead of raw mood score, at deviations from an individual’s average daily mood score. In order to do this, we’ll take each person’s average score and subtract each their scores from it, in order to find deviations from their mean.

Word clouds

Now that we’ve loaded the data, let’s go ahead and play around with it a little bit. We’re going to split the dataset up based on the standardized scores we just made with daily reported mood. We’ll break the dataset up into 2 groups: any text that was made on days where an individual’s mood was two stardard deviations above their mean (let’s call these positive days), and any text that was made on days where an individual’s mood was two standard deviation below their mean (let’s call these negative days). Now we can look at how texting patterns differ on positive and negative days. Let’s look at the most common words used on days when individuals report high overall mood.

Okay! It looks like we have words like “good,” “baby” and others that are good. This is a good sign so far. I wonder if the negative word cloud will be really different. Let’s check out what words are being used on days in which individuals report low daily mood.

Huh, well I think what’s interesting here is that there doesn’t seem to be much difference between the words being used on positive days and the words being used on negative days. You can still see words like “good” and “thank.” So it looks like simply breaking up word usage into positive days and negative days isn’t going to be enough for us to notice differences. We might have to think about how we can extract some signal from this noise.

As an aside, if it wasn’t clear, the population we’re collecting data from are high school students, and that is pretty well reflected in some of the more colorful word choices shown here.

Wourd counts

Now we’ve got a dataset of all of the words being typed. In addition, we also have information on how many times that any given word was used on positive days and on negative days. For example, we can see that ‘ew’ is used 3 times in our dataset, and all 3 uses are on negative mood days.

It might be good to get a feel for what are the most common used words on negative days and what are the most common used words on positive days.

While there are some unique words on the list, we can see that there are similar words on both lists, such as “like” and “wanna.” This tells us that these words are just words that are comomnly used, regardless of whether or not it’s a good or bad day.Now that we have a good sense of the words that are representative on positive and negative days, let’s take a look at the relationship between word usage on both of those days.

While it’s clear that there’s a positive relationship here, it’s hard to really tell because there are a lot of data points scrunched up in the lowe left quadrannt. Let’s do a quick and dirty log transformation of the data to get these points closer together. Not only will it make more visual sense, but there will be more space between the words. This means we can deploy a dynamic tool that allows us to display the word associated with a data point whenever we hover over it. Go ahead and put your cursor over the data points!

From a cursory glance, it looks like there is a positive relationship between positive words and negative words. This is indicating that even though there are words that are used a lot on negative days (such as ‘like’), they are also used a lot on positive days as well. That’s not really helpful for us if we’re wanting to find different word usages depending on daily mood. We’re more interested in the data points that would lie further away from a diagonal line going up and to the right. Let’s see if we can classify some of the words that appear more frequently on positive or negative days (as opposed to words that appear on both positive and negative days.

First, let’s simply get a measure for whether or not a word occurs on a positive day vs any day.

As we can see here, this is essentially showing us which words are most likely to be used on positive days. Essentially, based on this (very tentatively based), if we see the word zumiez, we can assume it is a positive day for that individual (again, let’s try too hard not to extrapolate out based on 6 uses of the word, but it’s a start).

But now we might be intersted in a similar metric but something slightly different. Before, we were looking at the likelihood (not actual likelihood, mind you) in which a word would appear on a positive day. But now let’s look at, on a positive day, what words are we most likely to see. Again, similar, but not necessarily the same.

Here we see a new list of words appear! Now we have words like “like,” “baby,” and “love.” The only problem is that words that are most likely to be found on positive days are not necessarily words that are only used on positive days. How can we reconcile these two things and potentially bring out some words that are both unique to positive days and likely to occur on positive days? One way we can do that is by using the harmonic mean. The harmonic mean is useful for this situation because the mean can be heavily influenced by outliers and larger numbers, while the harmonic mean penalizes outliers and can allow the transformation to be more sensitive to least elements within a list.

Unfortunately, it looks as though the harmonic mean didn’t produce much meaningfully different rankings of the data, so it doesn’t really add much to our understanding of the data. Another way of linking these two metrics is by using the cumulative distribution function (CDF). This wil tell us what the probability of a word occurring is. Essentially tell us the probability where the value of a word lies in the distributions for both the positive rate of the word and the positive frequency percentage of the word. This will put both terms onto a more similar scale, making comparing them (by taking the harmonic mean of the CDF values) significantly easier.

Well, as you are probably fairly aware, this is a very new list of words. It appears that the uniquely positive words that teenagers tend to use on positive days tend to be more sexual and romantic in nature. While it may be uncomfortable, it’s still important information for us to be aware of, and it’s not altogether unsurprising that individuals might feel better on days in which they are sexually active, or vice versa. From this table, we can now see the words that are most likely to show up on postive days and be specific to only positive days (pos_normcdf_hmean). Essentially, when there is a word with a high score on this, if we see this word being used, there’s a high chance that it’s a positive day, and if it’s a positive day, there’s a high probability that someone is using this word.

Let’s go ahead and run the same analyses for words on negative days.

Interestingly, we get words like “sorry” and “ok.” While the former might be obvious, it’s not unbelievable that individuals who write “ok” might be having a low mood day–especially given that it is a cultural meme that using “ok” in a text can be seen as standoffish or negative. Now that we’ve been able to pull some of the signal from the noise, let’s see if the word clouds are a little more meaninguf. Let’s start with the positive word cloud.

Great, we can already start to see some differences, like “good” and “thank.” Alright, let’s look at the negative word cloud now.

Well that’s more like it! There’s still some noise in there, but there are definitely things that we can pick out that make sense. In the positive word cloud we can see words like “good,” “happy,” and “want.” In the negative word cloud we see things like “sorry,” “bad,” “shit.” Now that we have this, let’s see if there’s another way that we can look at the relationship between the two types of data (positive and negative). I’ll create a scatter plot based on the positive and negative CDF harmonic means so we can see where the probability of each word falls in relationship to bad days and good days.

While the shape of the plot is relatively interesting, I think what’s more important are the values in the upper left quadrant and the lower right quadrant. Remeber, words in the upper left hand quadrant are the words that are most likely likely to happen on positive days and, when those words do show up, they’re likely to only occur on positive days. conversely, words in the lower right quadrant are the words that are most likely likely to happen on negative days and, when those words do show up, they’re likely to only occur on negative days.

If you hover your cursor over the points, you can see which word that point corresponds to. There are plenty of random words in here (“ezoo”), but there are clearly some interesting ones. Some positive words are “baby” and “sweetheart,” while some interesting negative words are “depressed” and “drunk.” Now that we’ve gotten a good look at how we can find differences in the data based on positive vs. negative days, it will be interesting if we can find associations in the data.

Best Fitting model

Now, the models that we’ve been running have kept the variables separated from one another. Let’s bring the models together and see which ones are strongest when we compare them directly. In the figure below, we’ve taken the within-person models from the three features above and plotted them. To understand this graph, you simply have to know that the further away something is from the center line (0), the larger the effect. And if the distribution crosses the center line, it’s not considered significantly different from 0. Here we can see that Word Count is the closest to the zero line–meaning that it has the smallest effect. In addition, it just barely crosses the zero line, which indicates that the effect is not statistically significant. The next furthest away variable is the FPP devations. We can see this is on the left side of the center line, meaning there is a negative effect, and since it doesn’t cross the center line, it is significantly different from 0. In essence, this is just another way of showing us what we already knew from above, that more FPP use is negatively related to daily mood. Finally, the furthest away from the center line is the sentiment variable. This is on the right side of the center line, meaning that there is a positive relationship! Positive mood is associated with positive word usage (at least when you say more positive things than you usually do).

Now this is all great, but what if we were to put all of these variables into ONE model. Would that be able to explain the variability in daily reported mood than any one variable by itself? If that’s so, how do we know which variables to choose? Not only that, but we can also see if the interaction between these variables might be meaningful (more on that in a second). One way we can do this is by running an algorithm that swaps out all of the terms and finds the model that explains the variability in mood the best. For time’s sake, I won’t show you that process, but I will tell you that the model that best explains the data is one with sentiment, FPP usage, and the interaction between these two. So what does that look like? Well, not much different from above, but now we know that it is the model that best explains the data. Take a look!

It’s now all the same color because these effects are all coming from the same model. Additionally, FPP and sentiment haven’t changed very much. There is a new term here, though, and that’s the one on bottom. This is the interaction between the two variables. It’s just barely not touching the zero line, so it’s not significant. But what exactly does it mean? Well, let’s pull up another figure to help explain it.

While this graph is confusing, one way we can interpret this graph is that when someone is +1 standard deviation above their average personal pronoun use, but also -2.5 standard deviations below their average sentiment, they’re more likely to be have a low reported daily mood. In other words, if you’re talking about yourself more, but using more negative words, you’re more likely to have a negative mood than if you’re talking aout yourself more but using positive words. In addition, you’re also more likely to have a better day if you’re using less first person pronouns, even if you’re using a lot of negative sentiment words.

Now this all has been great for understanding how some text features are related to daily mood. These types of models help us understand how phenomena are related–which is a useful endeavor. But what if we wanted to go beyond explaining and predict an individual’s mood.

Splitting the dataset

Now, notably, I used the same dataset to build the model as I did to predict it. Which is essentially the same as me predicting something in real life after it’s happened (which makes how poorly the model worked hurt all the more). A more appropriate way of going about this would be to build the model on one set of set of data and see how it performs on a completely separate set. One way we can do this is by simply randomly splitting our dataset into two different groups: one for training the model, and one for testing it.

However, because we have data that’s repeatedly measured across time, it wouldn’t be appropriate to randomly sample the data. In essence, we want past events predicting future ones, and not the other way around. So what we have to do with our dataset is take a certain portion of daily mood reports before a certain time, train a model on that, and test it on a model with data from after that time point. Another snag within this is that we should also be taking into account the fact that the data is grouped within a person too. So what we’re going to do is take the last 7 reported mood events for each person and put that into our testing set. Therefore, we will be training our model with everything before those data points, and testing it on the last 7 data points for each individual. This leaves us with a dataset of about 800 points for testing and 200 for training.

As you can see in the figure above, I’ve pulled out 9 random participants to show us how the data will be split. The red dots are what the model will be trained on, and the blue dots are what the model will be tested on. Let’s use the same model on the training set and then test it out on the test set (hopefully you’re not getting your hopes up). We can use this model as a benchmark to compare other models. In order to do that, we can use what’s know as the Root Mean Square Error (RMSE). This is simply a measure of how far off the predicted data points were off from the actual ones, where a higher number indicates a poorer prediction and a lower number indicates a better prediction.

It looks like the RMSE value is 19.586. Now we can use this to compare other models and see if we can get the number lower. But before we do that, let’s see if we can visualize the predicted vs actual points again for one participant.

Hmm, oh well. I guess the next step now is to see if we can predict these data points with some other type of models.

Conclusion

Overall, we were able to take a look at the variability in daily mood and we were able to look at how texting patterns differ on positive vs negative days. Additionally, we were able to examine the relationship between different text features and daily mood. We ultimately found that the within-subjects effects were much more striking and meaningful than the between-subject effects. However, unfortunately, just because there are relationships between these variables, that does not mean that we are close to being able to predict mood from the data we have. We’ll need plenty more data, and potentially even other features from the text data, in addition to other streams of passive sensing data (such as motion, GPS, light sensors). Once we incorporate all of these streams of data into a larger model, it’s possible that we might be able to get some better predictive power in our hands in order to do some real good for people.